TEX Live under Windows: what’s new with the 7th edition?

The 7th edition of TEX Live under Windows has some new features that are explained in this paper. Especially notable are two experiments to extend Kpathsea beyond its original abilities: • sharing Kpathsea data-structures between several processes for faster processing, • allow url’s in client programs as well as filenames whenever they ask Kpathsea to open a file. Additional points about the TeXSetup.exe program and the evolution of other parts of the distribution will also be discussed. Extensions to Kpathsea Rationale Kpathsea is at the heart of any Web2C based TEX distribution. The idea of gathering all the services needed by the TEX family programs around a common API was a great step forward, compared to the previous TEX distributions. Kpathsea features have been quite stable since version 3: there have been only minor changes. The syntax of the texmf.cnf file is the same, the file searching algorithm has converged to the current one, which may not be entirely satisfactory, but which is at least stable. One of the Web2C features – the \write18 command – triggered new uses of the whole TEX system. Specifically, it became possible to make tex call metapost, which will on its turn call tex again to typeset labels. As one document can hold many hundreds of metapost figures, we may wonder about the overhead when starting Kpathsea. In fact, we already noticed that the Kpathsea initialization time is far from insignificant. If we take the full TEX Live installation as a reference, then we have quite a long texmf.cnf file to parse and a huge ls-R hash-table to build, which are responsible for most of this initialization time. So for a given job, which needs to run tex and metapost in sequence for hundreds of figures, we can wonder if we could cut down the processing time by avoiding to repeat this initialization sequence: most of the data structures that Kpathsea will build across a single job will be the same for each instance of the programs. A second extant issue with Kpathsea is how far the path notion extends. From the beginning, Kpathsea was written to handle any kind of paths for any kind of operating systems: IBM MVS, DEC VMS, Unix, Amiga, DOS, OS/2, Win32. Not all of them are still actively supported, but they could be so in principle. Among them, Unix has probably the simplest, most regular path syntax. But supporting Win32 means that UNC share names like //Server/SharedDirectory/ have to be supported, as well as c:/Program Files/TeXLive/ texmf/tex/latex/base/article.cls. Looking at these pathname examples, we can see that each of them can be divided into 3 parts: a source (server, shared name, drive), a path relative to the source and the file name. So given that the URL syntax also follows the same pattern, we can wonder why it could not be supported by Kpathsea. A bare implementation of both features is active in the Win32 version of Kpathsea which is available on TEX Live 7. The description of their implementation and related problems follows. It is important to keep in mind that all the changes with report to the standard TEX Live sources have been made with simplicity in mind. Sometimes better solutions could have been found but at the price of more extensive changes or code rewriting. All of the TEX Live specific Win32 changes are provided by the /source/source-win32-patch.tar.bz2 file on the cdrom. 74 TUGboat, Volume 23 (2002), No. 1—Proceedings of the 2002 Annual Meeting TEX Live under Windows: what’s new with the 7th edition? Extending the “path” notion What do we want to achieve in extending Kpathsea towards the Internet? The basic usage could be something like this: \includegraphics{http://server.net/image.jpg} It could also be interesting for files included in a document to be taken from the Internet. We can wonder how long such a url might last (the Internet is moving so fast), but in the meantime more and more people are processing XML documents using TEX and so it can be useful to retrieve XML fragments off the Internet. Given this, we need to cope with only 3 kinds of url’s: file://, ftp://, http://. The others are not useful. The first one is not a problem because it is only syntactic sugar for local files. So we want at least that any file opened for reading by TEX can be replaced by an ftp:// or http:// url. What does this imply from the file point of view? Playing with remote files introduces unknown behaviour because the files may not be available. Moreover, both ftp and http protocols have different features. The http protocol allows for retrieving file information (availability, size) without transferring the actual file, whereas the ftp protocol cannot feasibly do this.1 The simplest solution for implementing remote file access is to use a third party library that downloads the file locally. Downloading the file at opening time will ensure future availability. All the remote files are given temporary names and the association between url’s and temporary names is stored in a hash-table. When Kpathsea exits, all such temporary files are removed. This is far from optimal, but safe enough for testing the feature. We could enhance the process by managing a cache of downloaded files, avoiding download of files that are already present unless they are too old and so on, but this is probably not a job for Kpathsea: if such a feature has to be used intensively, then it would be better to have a high-performance cache system for your whole Internet connection. The question is now the following one: where to hook in the function that will retrieve remote files? Given that we essentially expect TEX and friends to be the programs using remote files, we could hook into the web2c/lib/openclose.c file which holds the input/output functions for TEX programs. However there are also reasons to hook inside Kpathsea: • we need to extend path parsing and this is Kpathsea job; 1 Retrieving the file size requires parsing a directory listing, but there is no standard for this listing. • we can wonder about the opportunity of making Kpathsea Internet aware and be able to set up a remote texmf tree for example; • other programs than TEX engines could benefit from this feature; • under Win32, it is easier to replace kpathsea. dll if we want to enhance it or fix it. So we chose to make the changes inside Kpathsea rather than inside the TEX engines. The internals of Kpathsea have already a few macros and functions used to parse various kinds of paths, so adding a couple of macros and a few cases inside these functions was easy. Given the fact we want to download remote files as soon as they are accessed, we trap the fopen() call and hook file downloading there. This way, any Kpathsea file can be replaced by a url. This is enough to provide absolute remote file retrieval. However it is not enough yet to provide remote file searching. This would require to trap also the opendir() and readdir() calls. Although internet aware versions of these calls are provided by the Win32 WinInet API, this change has not been implemented yet2. The only remaining problem was to find a reliable library to download files. There are several choices: libwww the W3 Internet protocol library, which is Unix and Win32 compatible. After quick testing under Win32, file downloading was unreliable and blocked sometimes. Moreover, even after careful documentation reading, we have never found out how to display download progress, although it is said to be possible; libcurl this library is provided together with a command line tool and seemed to be very handy. It has been confirmed at least under Win32 where it worked like a charm at first try; wininet the Microsoft Internet library provided with Internet Explorer 5 and later. This is solid and has a high-level API, but Win32 only; sockets raw protocol handling is possible too but if we want reliability, better to rely on some higher level library. Ideally, it should be possible to plug any new library into the system and choose the one used from texmf.cnf. We need only one function to ensure the compatibility layer: int get_url_to_file ( /* the url of the file to download */ 2 In fact, because of the overhead introduced by initializing ls-R hash-tables for a remote texmf tree, this could be interesting in the larger framework of a Kpathsea server. See Karel Skoupy’s talk on this topic in this same proceedings. TUGboat, Volume 23 (2002), No. 1—Proceedings of the 2002 Annual Meeting 75